Computer Vision for Tissue Characterization and Outcome Prediction in Cancer

The aim of this dissertation was to investigate the use of computer vision for tissue characterization and patient outcome prediction in cancer. This work focused on analysis of digitized tissue specimens, which were stained only for basic morphology (i.e. hematoxylin and eosin). The applicability of texture analysis and convolutional neural networks was evaluated for detection of biologically and clinically relevant features. Moreover, novel approaches to guide ground-truth annotation and outcome-supervised learning for prediction of patient survival directly from the tumor tissue images without expert guidance was investigated. We first studied quantification of tumor viability through segmentation of necrotic and viable tissue compartments. We developed a regional texture analysis method, which was trained and tested on whole sections of mouse xenograft models of human lung cancer. Our experiments showed that the proposed segmentation was able to discriminate between viable and non-viable tissue regions with high accuracy when compared to human expert assessment. We next investigated the feasibility of pre-trained convolutional neural networks in analysis of breast cancer tissue, aiming to quantify tumor-infiltrating lymphocytes in the specimens. Interestingly, our results showed that pre-trained convolutional neural networks can be adapted for analysis of histological image data, outperforming texture analysis. The results also indicated that the computerized assessment was on par with pathologist assessments. Moreover, the study presented an image annotation technique guided by specific antibody staining for improved ground-truth labeling. Direct outcome prediction in breast cancer was then studied using a nationwide patient cohort. A computerized pipeline, which incorporated orderless feature aggregation and convolutional image descriptors for outcome-supervised classification, resulted in a risk grouping that was predictive of both disease-specific and overall survival. Surprisingly, further analysis suggested that the computerized risk prediction was also an independent prognostic factor that provided information complementary to the standard clinicopathological factors. This doctoral thesis demonstrated how computer-vision methods can be powerful tools in analysis of cancer tissue samples, highlighting strategies for supervised characterization of tissue entities and an approach for identification of novel prognostic morphological features.Kudosnäytteiden mikroskooppisten piirteiden visuaalinen tarkastelu on yksi tärkeimmistä määrityksistä syöpäpotilaiden diagnosoinnissa ja hoidon suunnittelussa. Edistyneet kuvantamisteknologiat ovat mahdollistaneet histologisten kasvainkudosnäytteiden digitalisoinnin tarkalla resoluutiolla. Näytteiden digitalisoinnin seurauksena niiden analysointiin voidaan soveltaa edistyneitä koneoppimiseen perustuvia konenäön menetelmiä. Tämä väitöskirja tutkii konenäön menetelmien soveltamista syöpäkudosnäytteiden laskennalliseen analyysiin. Työssä tutkitaan yksittäisten histologisten entiteettien, kuten nekroottisen kudoksen ja immuunisolujen automaattista kvantifiointia. Lisäksi työssä esitellään menetelmä potilaan selviytymisen ennustamiseen pelkkään kudosmorfologiaan perustuen

Self-Supervised Learning of Phenotypic Representations from Cell Images with Weak Labels

We propose WS-DINO as a novel framework to use weak label information in learning phenotypic representations from high-content fluorescent images of cells. Our model is based on a knowledge distillation approach with a vision transformer backbone (DINO), and we use this as a benchmark model for our study. Using WS-DINO, we fine-tuned with weak label information available in high-content microscopy screens (treatment and compound), and achieve state-of-the-art performance in not-same-compound mechanism of action prediction on the BBBC021 dataset (98%), and not-same-compound-and-batch performance (96%) using the compound as the weak label. Our method bypasses single cell cropping as a pre-processing step, and using self-attention maps we show that the model learns structurally meaningful phenotypic profiles

Identification of tumor epithelium and stroma in tissue microarrays using texture analysis

<p>Abstract</p> <p>Background</p> <p>The aim of the study was to assess whether texture analysis is feasible for automated identification of epithelium and stroma in digitized tumor tissue microarrays (TMAs). Texture analysis based on local binary patterns (LBP) has previously been used successfully in applications such as face recognition and industrial machine vision. TMAs with tissue samples from 643 patients with colorectal cancer were digitized using a whole slide scanner and areas representing epithelium and stroma were annotated in the images. Well-defined images of epithelium (n = 41) and stroma (n = 39) were used for training a support vector machine (SVM) classifier with LBP texture features and a contrast measure C (LBP/C) as input. We optimized the classifier on a validation set (n = 576) and then assessed its performance on an independent test set of images (n = 720). Finally, the performance of the LBP/C classifier was evaluated against classifiers based on Haralick texture features and Gabor filtered images.</p> <p>Results</p> <p>The proposed approach using LPB/C texture features was able to correctly differentiate epithelium from stroma according to texture: the agreement between the classifier and the human observer was 97 per cent (kappa value = 0.934, <it>P </it>< 0.0001) and the accuracy (area under the ROC curve) of the LBP/C classifier was 0.995 (CI95% 0.991-0.998). The accuracy of the corresponding classifiers based on Haralick features and Gabor-filter images were 0.976 and 0.981 respectively.</p> <p>Conclusions</p> <p>The method illustrates the capability of automated segmentation of epithelial and stromal tissue in TMAs based on texture features and an SVM classifier. Applications include tissue specific assessment of gene and protein expression, as well as computerized analysis of the tumor microenvironment.</p> <p>Virtual slides</p> <p>The virtual slide(s) for this article can be found here: <url>http://www.diagnosticpathology.diagnomx.eu/vs/4123422336534537</url></p

An open-source, MATLAB based annotation tool for virtual slides

From 11th European Congress on Telepathology and 5th International Congress on Virtual Microscopy Venice, Italy. 6-9 June 2012Peer reviewe

Quantification of Estrogen Receptor-Alpha Expression in Human Breast Carcinomas With a Miniaturized, Low-Cost Digital Microscope : A Comparison with a High-End Whole Slide- Scanner

Introduction: A significant barrier to medical diagnostics in low-resource environments is the lack of medical care and equipment. Here we present a low-cost, cloud-connected digital microscope for applications at the point-of-care. We evaluate the performance of the device in the digital assessment of estrogen receptor-alpha (ER) expression in breast cancer samples. Studies suggest computer-assisted analysis of tumor samples digitized with whole slide-scanners may be comparable to manual scoring, here we study whether similar results can be obtained with the device presented. Materials and methods: A total of 170 samples of human breast carcinoma, immunostained for ER expression, were digitized with a high-end slide-scanner and the point-of-care microscope. Corresponding regions from the samples were extracted, and ER status was determined visually and digitally. Samples were classified as ER negative (<1% ER positivity) or positive, and further into weakly (1-10% positivity) and strongly positive. Interobserver agreement (Cohen's kappa) was measured and correlation coefficients (Pearson's product-momentum) were calculated for comparison of the methods. Results: Correlation and interobserver agreement (r = 0.98, p < 0.001, kappa = 0.84, CI95% = 0.75-0.94) were strong in the results from both devices. Concordance of the point-of-care microscope and the manual scoring was good (r = 0.94, p < 0.001, kappa = 0.71, CI95% = 0.61-0.80), and comparable to the concordance between the slide scanner and manual scoring (r = 0.93, p < 0.001, kappa = 0.69, CI95% = 0.60-0.78). Fourteen (8%) discrepant cases between manual and device-based scoring were present with the slide scanner, and 16 (9%) with the point-of-care microscope, all representing samples of low ER expression. Conclusions: Tumor ER status can be accurately quantified with a low-cost imaging device and digital image-analysis, with results comparable to conventional computer-assisted or manual scoring. This technology could potentially be expanded for other histopathological applications at the point-of-care

Systems pathology by multiplexed immunohistochemistry and whole-slide digital image analysis

The paradigm of molecular histopathology is shifting from a single-marker immunohistochemistry towards multiplexed detection of markers to better understand the complex pathological processes. However, there are no systems allowing multiplexed IHC (mIHC) with high-resolution whole-slide tissue imaging and analysis, yet providing feasible throughput for routine use. We present an mIHC platform combining fluorescent and chromogenic staining with automated whole-slide imaging and integrated whole-slide image analysis, enabling simultaneous detection of six protein markers and nuclei, and automatic quantification and classification of hundreds of thousands of cells in situ in formalin-fixed paraffin-embedded tissues. In the first proof-of-concept, we detected immune cells at cell-level resolution (n = 128,894 cells) in human prostate cancer, and analysed T cell subpopulations in different tumour compartments (epithelium vs. stroma). In the second proof-of-concept, we demonstrated an automatic classification of epithelial cell populations (n = 83,558) and glands (benign vs. cancer) in prostate cancer with simultaneous analysis of androgen receptor (AR) and alpha-methylacyl-CoA (AMACR) expression at cell-level resolution. We conclude that the open-source combination of 8-plex mIHC detection, whole-slide image acquisition and analysis provides a robust tool allowing quantitative, spatially resolved whole-slide tissue cytometry directly in formalin-fixed human tumour tissues for improved characterization of histology and the tumour microenvironment.Peer reviewe

Deep learning based tissue analysis predicts outcome in colorectal cancer

Image-based machine learning and deep learning in particular has recently shown expert-level accuracy in medical image classification. In this study, we combine convolutional and recurrent architectures to train a deep network to predict colorectal cancer outcome based on images of tumour tissue samples. The novelty of our approach is that we directly predict patient outcome, without any intermediate tissue classification. We evaluate a set of digitized haematoxylin-eosin-stained tumour tissue microarray (TMA) samples from 420 colorectal cancer patients with clinicopathological and outcome data available. The results show that deep learning-based outcome prediction with only small tissue areas as input outperforms (hazard ratio 2.3; CI 95% 1.79-3.03; AUC 0.69) visual histological assessment performed by human experts on both TMA spot (HR 1.67; CI 95% 1.28-2.19; AUC 0.58) and whole-slide level (HR 1.65; CI 95% 1.30-2.15; AUC 0.57) in the stratification into low-and high-risk patients. Our results suggest that state-of-the-art deep learning techniques can extract more prognostic information from the tissue morphology of colorectal cancer than an experienced human observer.Peer reviewe

CDX2 Loss With Microsatellite Stable Phenotype Predicts Poor Clinical Outcome in Stage II Colorectal Carcinoma

Current risk factors in stage II colorectal carcinoma are insufficient to guide treatment decisions. Loss of CDX2 has been shown to associate with poor clinical outcome and predict benefit for adjuvant chemotherapy in stage II and III colorectal carcinoma. The prognostic relevance of CDX2 in stage II disease has not been sufficiently validated, especially in relation to clinical risk factors, such as microsatellite instability (MSI) status, BRAF mutation status, and tumor budding. In this study, we evaluated the protein expression of CDX2 in tumor center and front areas in a tissue microarrays material of stage II colorectal carcinoma patients (n=232). CDX2 expression showed a partial or total loss in respective areas in 8.6% and 10.9% of patient cases. Patients with loss of CDX2 had shorter disease-specific survival when scored independently either in tumor center or tumor front areas (log rank P=0.012; P=0.012). Loss of CDX2 predicted survival independently of other stage II risk factors, such as MSI status and BRAF mutation status, pT class, and tumor budding (hazard ratio=5.96, 95% confidence interval=1.55-22.95; hazard ratio=3.70, 95% confidence interval=1.30-10.56). Importantly, CDX2 loss predicted inferior survival only in patients with microsatellite stable, but not with MSI-high phenotype. Interestingly, CDX2 loss associated with low E-cadherin expression, tight junction disruption, and high expression of ezrin protein. The work demonstrates that loss of CDX2 is an independent risk factor of poor disease-specific survival in stage II colorectal carcinoma. Furthermore, the study suggests that CDX2 loss is linked with epithelial-to-mesenchymal transition independently of tumor budding.Peer reviewe

Combined epithelial marker analysis of tumour budding in stage II colorectal cancer

Tumour budding predicts survival of stage II colorectal cancer (CRC) and has been suggested to be associated with epithelial-to-mesenchymal transition (EMT). However, the underlying molecular changes of tumour budding remain poorly understood. Here, we performed multiplex immunohistochemistry (mIHC) to phenotypically profile tumours using known EMT-associated markers: E-cadherin (adherence junctions), integrin beta 4 (ITGB4; basement membrane), ZO-1 (tight junctions), and pan-cytokeratin. A subpopulation of patients showed high ITGB4 expression in tumour buds, and this coincided with a switch of ITGB4 localisation from the basal membrane of intact epithelium to the cytoplasm of budding cells. Digital image analysis demonstrated that tumour budding with high ITGB4 expression in tissue microarray (TMA) cores correlated with tumour budding assessed from haematoxylin and eosin (H&E) whole sections and independently predicted poor disease-specific survival in two independent stage II CRC cohorts (hazard ratio [HR] = 4.50 (95% confidence interval [CI] = 1.50-13.5), n = 232; HR = 3.52 (95% CI = 1.30-9.53), n = 72). Furthermore, digitally obtained ITGB4-high bud count in random TMA cores was better associated with survival outcome than visual tumour bud count in corresponding H&E-stained samples. In summary, the mIHC-based phenotypic profiling of human tumour tissue shows strong potential for the molecular characterisation of tumour biology and for the discovery of novel prognostic biomarkers.Peer reviewe